5 Future Work 4.2 Learning with an Arbitrary Output Unit: Max K-f Consistency

agnostic learning of neural networks with bounded fan-in. It would be interesting to extend the hardness result for networks with real outputs to the case of a linear output unit with a constraint on the size of the output weights. We conjecture that a similar result can be obtained, with a relative error bound that|unlike Vu's result for this case 19]|does not decrease as the input dimension increases. It would also be worthwhile to extend the results to show that it is diicult to nd a hypothesis that has expected loss nearly minimal over some neural network class, whatever hypothesis class is used. There is some related work in this direction. Theorem 7 in 3] shows that nding a conjunction of k 0 linear threshold functions that correctly classiies a set that can be correctly classiied by a conjunction of k linear threshold functions is as hard as colouring a k-colourable graph with n vertices using k 0 colours, which has since been shown to be hard for k 0 = O(kn) for some > 0 15]. The cryptographic results mentioned in Section 1 do not have such strong restrictions on the hypothesis class, but only apply to classes that are apparently considerably richer than the neural network classes studied in this paper. It is easy to show that the result is also true if the components of vectors in S in and S out must be ratios of integers, provided the integers are allowed to be as large as ck 2 , for some constant c. Hence, for each k, the number of bits needed to represent S is linear in the size of the graph G. These lemmas show that we have an L-reduction from Max k-Cut for k-colourable graphs to Max k-F Consistency, with parameters = k=(k ? 1) and = 9(4k + 1), and this L-reduction preserves maximality. Combining this with Theorem 5 gives Theorem 2. 4.3 Learning with a sigmoid output unit: Max k-Consistency We give an L-reduction from Max k-F Consistency to Max k-Consistency , where F is the class of linear threshold functions. Given a sample S for a Max k-F Consistency problem, we use the same sample for the Max k-Consistency problem. Trivially 1 , if opt Max k-F Consistency (S) = 1 then opt Max k-Consistency (S) = 1. Furthermore, we have the following lemma. Lemma 7. For a solution f …